System for hypervideo filtering based on end-user payment interest and capability

ABSTRACT

A system for selling digital video information over a communications network such as the Internet basically includes a filtering server and a service administration system. The filtering server includes a commerce server, a streaming server and a world-wide web information server. The world-wide web information server provides customers with a list of available videos, the possibility to customize specific versions and a method for calculating the price for a specific version. The service administration system includes a management tool for the merchant to define price classes and payment schemes and a meta-data editing tool for the merchant to prepare the content for the filtering. The system will connect to external systems to have the payment information verified. If the payment transaction is accepted the system will pass the selected video through a sequence of service filters to generate the edition that the customer selected.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to hypervideo filtering and moreparticularly to providing customers access through a communicationsnetwork to digital video information stored on a merchant's server.

[0003] 2. Description of the Prior Art

[0004] Electronic commerce over communications networks is becoming animportant technology in today's market place. Electronic commerce allowsgoods and services to be sold over a network and also facilitates thesales and distribution of digital information. At the same time, digitalvideo storage and streaming (i.e., delivery over a network) is becominga commodity.

[0005] Video is a powerful media for documentation, informationexchange, and, of course, for entertainment. In a similar way, globalcommunications networks such as the Internet constitute powerfulchannels for distributing information to customers. Mostly, Internetusers have been able to consume the information for free. For contentowners, this shopping model is not attractive as it gives fewopportunities for increased revenues. There are some content owners andcontent brokers connected to the Internet today, but only a few givedirect access to the information objects. The Internet Movie Database(http://us.imdb.com/), E-online's Movie Finder(http://www.moviefinder.com/), and the Vanderbilt University'sTelevision News Archive (http:/tvnews.vanderbilt.edu/) are examples ofvideo archives that can be accessed over the Internet. None of theseservices gives customers a means to access to the video objects directlytoday. It is an object of the present invention to provide a system forcontent owners or brokers to become merchants that can sell videoinformation over a network.

[0006] It will be important to such video information merchants to beable to combine video information into several products that can be soldat different prices. A television news merchant, for instance, may wantto sell four products: Headline News, International News, National News,and Complete News coverage that are all based on the merchant'srepository of video information but provide different views and depth ofcoverage. The value of each of these products may be different so themerchant may want to be able to price them differently.

[0007] It is also important for the merchant to be able to provide thisservice without prior agreements with each customer individually. Thecustomer's ability to pay should be verified on a single transactionbasis and the payment transaction should be completed immediately. Withthis approach, the merchant can provide such services globally and havethe product immediately available at the market place.

[0008] In many cases, the video information merchant may want to providecross-references or hyperlinks between information units in the archive.Hypervideo facilitates such hyperlinking.

[0009] Hypervideo, which is described by N. Sawhney, D. Balcom, and I.Smith in “Authoring And Navigating Video In Space And Time” , IEEEMultimedia 4(4), 30-39, October, 1997 and by W.-h. Ma, Y.-J. Lee, D. H.C. Du and M. P. McCahill in “Video-based Hypermedia ForEducation-on-demand”, IEEE Multimedia 5(1), January, 1998, is anon-linear type of video. Hypervideo is consisting of story units calledNarrative Sequences of Scenes—NSS (adopted from [Sawhney et al. 1997])and hyperlinks connecting these story units together. The source anddestination node for a hyperlink may be called an Anchorable InformationUnit (AIU). In a hypervideo, there are many different types of AIUs. ANSS may itself be an AIU. A temporal or a spatio-temporal region of aNSS may be an AIU (in this case often called hotspot). Any non-videoobjects such as portions of a hypertext object, pictures, or charts mayalso be AIUs.

[0010] A hypervideo may have several parallel tracks where informationis streamed synchronously. For instance, a hypervideo presentation maycontain three parallel tracks; one for the video itself, one for atextual transcript of what is being said, and one for an image of thecurrent narrator. Also, there might be several alternative data streamsthat can be fed into one track. There might be many differentsub-titles, for instance, to accomplish multi-lingual sub-titling.

[0011] The process of offering different views of hypervideo informationcan be seen as a filtering process. Thus, the research work done in thearea of customized video services by G. Ahanger and T. D. C. Little in“A System For Customized News Delivery From Video Archives” ,Proceedings Of The 4^(th) International Conference On MultimediaComputing And Systems, Ottawa, Canada, June 1997 and as described inU.S. Pat. No. 5,434,678 entitled “Seamless Transmission OfNon-sequential Video Segments” issued to M. Abecassis on Jul. 18 1995,is relevant for providing video information services over a network.This prior art, however, does not address the issues of hypervideofiltering, for instance filtering of AIUs and hyperlinks. Also, thefocus for this prior art is to find the most appropriate way to indexthe semantic contents of video information and to rank informationobjects based on how relevant these are according to an end-userinterest profile. A video information merchant may not need to, or wantto know the end-user interest. Thus, it is an object of the presentinvention to provide filtering that is based on the end-user's paymentchoice and capability. This commercial aspect of information filteringhas not been addressed by the prior art.

[0012] Most of the work in the area of electronic commerce is devoted togenerating catalogs from a database of product information, to definingappropriate shopping models and to provide secure payment mechanisms.There is, however, one patent related to the distribution of digitalcontent. This patent is U.S. Pat. No. 5,646,992 entitled “Assembly,Distribution, and Use Of Digital Information” , issued to R. J. Sublerand T. M. Hastings on Jul. 8, 1997 and assigned to Digital Delivery,Inc. In this system, an encryption scheme using multiple decryption keysis used to give customers access to a subset of items. Each decryptionkey will permit decryption of all items belonging to a specific subset.

SUMMARY OF THE INVENTION

[0013] The present invention provides a system for selling digital videoinformation over a communications network such as the Internet. Theinformation is stored on an information server at a merchant's videoinformation site. Videos are represented as hypervideos consisting ofnarrative sequences and hyperlinks. Hyperlinks connect anchorableinformation units within the hypervideo to other anchorable informationunits within the same hypervideo or to anchorable information unitsoutside of the hypervideo. A video information customer may establish anetwork connection to the merchant to browse or search a catalog ofavailable video information. When ordering a video, the customer willhave several editions of the same video to choose among. Each editionmay have a different quality, duration and price than other editions.When the customer selects a given video edition, the customer will alsoprovide the system with electronic payment information, such as creditcard information.

[0014] The system basically comprises a filtering server and a serviceadministration system. The filtering server comprises a commerce serverfor initiating electronic payment transactions, a streaming server thatdelivers purchased videos to the respective customers and a world-wideweb information server. The world-wide web information server providescustomers with the following; a list of available videos, thepossibility to customize specific versions and a method for calculatingthe price for a specific version. The world-wide web information serveralso allows the end-user to electronically purchase a specific versionand upon positive outcome of an electronic payment transaction,generates the specific version based on multimedia data and descriptivedata stored in the system. The world-wide web further returns thereference to the specific version to the customer for viewing. Theservice administration system comprises a management tool for themerchant to define price classes and payment schemes according to whichthe information can be priced and a meta-data editing tool for themerchant to prepare the content for the filtering by defininghyperlinks, by categorizing video nodes and hyperlinks according toprice classes, and by defining the amount of promotional information toinsert and the rules used to control insertion of promotionalinformation.

[0015] The system will connect to external systems to have the paymentinformation verified. If the payment transaction is accepted the systemwill pass the selected video through a sequence of service filters togenerate the edition that the customer selected. A reference to thespecific edition of the video is then transferred back to the customerto establish a video transfer connection between the customer's videodisplay unit and the video information merchant's video server.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016]FIG. 1 illustrates a block diagram of the present invention.

[0017]FIG. 2 illustrates notation convention for hypervideo sequences.

[0018]FIG. 3 illustrates a first step of the present invention which isdigitizing a video tape.

[0019]FIG. 4 illustrates a second step of the present invention which isdetermining video shots, scenes and narrative sequences.

[0020]FIG. 5 illustrates a third-step of the present invention which isgenerating meta-data and building up a hypervideo structure.

[0021]FIG. 6 illustrates a fourth step of the present invention which isfiltering meta-data and adding payment schemes and values.

[0022]FIG. 7 illustrates a detailed step within filtering which isselection of Narrative Sequences of Scenes (NSSs) or videos to beviewed.

[0023]FIG. 8 illustrates a detailed step within filtering which iscalculating one or more prices according to payment or customer profile.

[0024]FIG. 9 illustrates a detailed step within filtering which isperforming the payment transaction.

[0025]FIG. 10 illustrates an example of the results of filtering.

[0026]FIG. 11 illustrates a detailed optional step within filteringwhich is inserting a promotional video.

[0027]FIG. 12 illustrates a detailed step within filtering which iscopying sequences and AIU's to streaming server.

[0028]FIG. 13 illustrates a detailed step within filtering which isinitiating the streaming process.

[0029]FIG. 14 illustrates a detailed optional step within filteringwhich is monitoring the streaming process.

[0030]FIG. 15 illustrates a detailed step within filtering which isinitiating and starting another streaming process.

[0031]FIGS. 16 and 17 illustrate multidimensional hyperlinking usingAIUs.

[0032]FIG. 18 illustrates version branching using AIUs.

[0033]FIG. 19 illustrates a detailed optional step within filteringwhich is reimbursing the customer for hypervideo elements not consumed.

DETAILED DESCRIPTION OF THE INVENTION

[0034]FIG. 1 illustrates a block diagram of the present invention. Thefiltering system of the present invention needs access to threedifferent data repositories. These repositories may be stored separatelyor may be handled by one, common data manager. Media data repository 11stores multimedia objects such as text documents, images, audio, andvideo shots. Meta-data repository 12 stores the properties (encoding,location, etc. . . ) of media objects and descriptions of the contentsof multimedia objects. Media data repository 11 and meta-data repository12 could be contained within filtering server 10, could be containedexternally within archiving system 13, or could be contained in twoseparate locations. Filtering meta-data repository 14 contains filteringmeta-data and payment schemes. These repositories are described in moredetail below.

[0035] Within media data repository 11, Narrative Sequences of Scenes(NSSs) are represented in a digital video file format (for instance inthe RealMedia streaming format). Each NSS may be stored as a separatefile, or several NSSs may be stored contiguously in one file in thedefault sequence. Media data repository 11 and media data server 15 aregeneral components that do not store or handle any filteringinformation. For security reasons, however, media data server 15 andmedia data repository 11 may consist of two parts; one part which isaccessible for streaming server 16 (and, consequently, accessible by theend-user through streaming client 22) and one part which is onlyaccessible for internal components such as media file generator 17.

[0036] Media data repository 11 also contains other multimedia objects.Non-video objects may define temporal data streams (for instance audiotracks or closed captioning) or static objects (such as still images andtext documents). The output from media file generator 17 is one or moremedia streaming files where various types of media objects have beenmerged together. Alternatively, the output from media file generator 17is one or more script files containing a specification of the resultingmultimedia presentation—e.g., in SMIL format (specified in “SynchronizedMultimedia Integration Language (SMIL) 1.0 Specification”, W3C, Jun. 15,1998—http:/www.w3.org/TR/1998/REC-smil-19980615. These files will bestored in media data repository 11.

[0037] A hypervideo composition meta-data can be represented as a graphof AIUs interconnected by hyperlinks. Meta-data repository 12 storesthese graphs as AIU objects and hyperlink objects where hyperlinkobjects connect a source AIU to one destination AIU. Several hyperlinksmay share source AIUs and destination AIUs. Meta-data repository 12 mayalso contain content indices as described in U.S. patent applicationSer. No. 09/053,761 entitled “System And Methods For ProvidingStructured Logging, Querying And Navigation For Multimedia Management” ,filed on Apr. 2, 1998, assigned to the same assignees as the presentinvention and hereby incorporated by reference.

[0038] For payment meta-data, the filtering service of the presentinvention uses one or more filtering parameters for filtering AIUs andhyperlinks. A filtering parameter is a special type of content indicesthat is used by the filtering service.

[0039] For payment schemes, the filtering meta-data repository 14defines one or more filtering services and stores the payment schemesthat will be used by each of these services. The repository 14 alsodefines a number of price classes and associates each class with a givenprice. The information used to represent the individual schemes is:1)Subscription schemes where various subscription levels are defined.Each subscription level is given a name and is associated with a priceclass. Also, each subscription level defines which values individualfiltering parameters are to take when filtering video for customerssubscribing to this level of service and it defines the amount ofpromotional video insert that can be inserted (in fraction of totalplayback duration). 2)Fixed price per hypervideo where various versiontypes are defined. Each version type is given a name and is associatedwith a price class. Also each version type defines which valuesindividual filtering parameters are to take when generating a specificversion of a specific hypervideo and it defines the amount ofpromotional video insert that can be inserted (in fraction of totalplayback duration). 3)Price calculated from the composed video whereeach type of information object is associated with a triple of priceclasses as <static_price, component_factor, duration_factor> allowingeach value to be negative, positive, or zero. The static_price classgives the cost of the given information object. The component_factor isused to calculate the price of a given object as being set to theaccumulated price of all of the object's components multiplied by thefactor. For instance, if a news video contains four news stories—eachhaving a static_price that is mapped to the value of a quarter of adollar, and the component_factor is set to one, the price of the newsvideo is one dollar. The duration_factor is used to calculate the priceof a given object by multiplying the playback duration of the objectwith the factor. For instance, if a news story lasts for two and a halfminutes and the duration_factor is set to ten cents per minute, theprice of the-news story will be a quarter of a dollar.

[0040] Media file generator 17 and filtering process 18 are containedwithin www server 20. Filtering process 18 is responsible for presentingthe appropriate choices to the end-user via www client 24, for computingthe price of the selected version of hypervideo, for initiating apayment transaction and for generating a specific filtered version.Filtering process 18 requests meta-data from database server 21 whichmanages filtering meta-data DB 14 and which interfaces with archivingsystem 13.

[0041] Media file generator 17 takes the input from filtering process 18to generate file(s) storing the data stream(s) representing the filteredversion of the hypervideo. Media file generator 17 may output mediafiles created by cutting and pasting segments from the input files.Alternatively, if supported by streaming server 16, media file generator17 may generate script files (e.g., SMIL files) instructing streamingserver 16 how to create the output stream at streaming time. Media filegenerator 17 retrieves necessary meta-data from database server 21 andmedia data (if needed) from media data server 15. Output files are sentto media data server 15 and can later be read by streaming server 16.

[0042] For media streaming, streaming server 16 and streaming client 22are responsible for presenting media data stored on media server 15 tothe customer via streaming client 22. For payment, wallet 25, commerceserver 26, and Secure Electronic Transaction (SET) server 27 areresponsible for initiating a payment transaction and exchanging paymentinformation with payment gateway 28 of clearing house 29.

[0043] The merchant uses administrative tools 32 within serviceadministrative system 30, which interfaces with filtering server 10, todefine filtering services and payment schemes. Meta-data editing tool 33allows the merchant to define NSSs and AIUs and to associate filteringmeta-data with such elements.

[0044] The present invention utilizes different types of data toaccomplish the filtering by payment. The notation convention forhypervideo sequences used throughout this description is shown in FIG.2. Left-hand part 40 of FIG. 2 illustrates how the different types ofdata relate to each other. As can be seen from FIG. 2, this way ofillustrating a hypervideo along with its meta-data tends to be rathercomplex. To allow a simplified representation of hypervideo sequences,the notation shown on right-hand part 41 of FIG. 2 will also be used foran equivalent representation of sequences of scenes of a hypervideo.

[0045] The left part 40 of FIG. 2 illustrates the different types ofdata that are being used by the filtering system of the presentinvention. Top level 42 consists of the individual narrative sequencesin the hypervideo object. Each object (a, b, c, etc.) at this level 42represents a collection of consecutive video frames (media data). Videometa-data layer 43 describes the narrative sequences and specifies theoriginal sequencing of NSSs (A, B, C, etc.). It is assumed that thesetwo layers 42 and 43, are outside of the control of the filteringsystems, as for instance, the case if the video is stored on a DVD.Filtering meta-data layer 44 contains the meta-data being used by thefiltering system. This includes sequencing information (solid linesbetween filtering meta-data units), identification of anchorableinformation units and hyperlinks (dashed lines between filteringmeta-data units), and parameters associated with narrative sequences andhyperlinks for filtering and payment purposes. Filtering meta-data layer44 also contains references to the next layer 45 where the paymentobjects and the payment scheme are defined.

[0046] The following describes how hypervideo should be analyzed andindexed to generate the filtering meta-data needed. This is performed inservice administration system 30 of FIG. 1. The first three steps areonly needed if the video to be indexed is stored on a video tape oranother medium without direct access to individual shots or scenes. Insome cases, the video is already segmented and indexed. This might bethe case for videos stored on DVDs and for videos stored in a digitalarchive as, for instance, described by Hjelsvold et al. 1998.

[0047]FIG. 3 illustrates the first step of the present invention whichis the digitizing of video tape 46. The video is digitized to generate astream of raw video data 47 that can be further processed.

[0048]FIG. 4 illustrates the second step of the present invention whichis that video shots, scenes and narrative sequences of scenes 50 aredetermined. Individual shots, scenes, and narrative sequences of scenes50 have to be determined because these are the building blocks indefining the hypervideo units to be filtered. This can either be donemanually, automatically by a video processing system, orsemi-automatically in a hybrid system. This is described by R.Depommier, N. P. Fan, K. Gunaseelan, R. Hjelsvold, S.-P. Liou and E.Stiller-Erdpresser in “CARAT-ARC: A Scaleable And Reliable Digital MediaArchiving System”, Proceedings Of The IEEE International Conference OnImage Processing, 1997.

[0049]FIG. 5 illustrates how meta-data 51 are generated and how ahypervideo structure is built up. Meta-data 51 are generated to definethe NSS sequencing, to identify the AIUs, and to add hyperlinks betweenthese AIUs. This step is not necessary when the hypervideo represents atraditional video with sequential ordering of NSSs in the order theyappear in the input video and with no AIUs and hyperlinks defined. Inthis case, meta-data 51 will only define the standard sequencing of theNSSs. When this step is completed, digital video content (NSSs) 52 willbe stored in a digital video repository 53 (11 of FIG. 1) while videometa-data 51 specifying NSS sequencing and defining AIUs are stored inmeta-data database 54 (12 of FIG. 1). Repository 53 and database 54 maybe managed by the same database system, may be collocated on the samecomputer system but managed by different systems, or may be stored andmanaged on separate computer systems.

[0050]FIG. 6 illustrates the filtering of meta-data and the adding ofpayment schemes and values. Filtering meta-data 60, payment meta-data 61and video meta-data units 62 are associated with filtering and payment.In this step the sequencing of NSSs and the definition of AIUs from FIG.5 can be extended and modified, automatically or manually. Also, theassociations between these units 63 and the payment information areestablished during this step. Digital video repository 64, meta-datadatabase 65 and filtering meta-data database 66 are also shown.

[0051] A buying customer may establish a connection to the hypervideomerchant's server to buy hypervideo information. The following willdescribe how the video is filtered by filtering server, 10 of FIG. 1,according to a user's price selection, how an electronic paymenttransaction is integrated, and how delivery of the filtered hypervideois initiated and monitored. A detailed description of filtering andpayment meta-data is given below.

[0052]FIG. 7 illustrates the first step in filtering, payment anddelivery which is the selection of NSSs or videos to be viewed. Thepresent invention defines two main approaches that can be used by thecustomer to select the NSSs or videos to be viewed. First, the customermay browse a catalog of available hypervideo. Alternatively, thecustomer may make a query for NSSs associated with a certain topic. Thecustomer may now make a selection among the NSSs the customer isauthorized to view based on his or her interest. The customer'sselection represents a hypervideo 70 which includes all potentialvideos, scenes (NSSs) and hyperlinking (AIUs) that were selected. Allsequences can be selected and streamed. This content is physicallyavailable on appropriate media (conventional hard drives, DVDs ROMs,memory or storage devices of any type that can be accessed by one orseveral workstations or servers). The corresponding meta-data is beingused by subsequent filtering and payment computation steps.

[0053]FIG. 8 describes the second step, how one or multiple prices arecalculated according to payment or customer profile. The filteringservice may define several payment schemes. Such schemes are discussedin more detail below. These schemes are now analyzed together with thefiltering meta-data to present to the user a price choice. Assume, forinstance, that the price calculation process in this example identifiedthree different combinations of hypervideo sequences (NSSs) andcalculated the relevant prices. The customer then selects one videocombination according to the given prices and selects the preferredcredit card. This is illustrated in FIG. 8.

[0054] In the next step, step 3, the customer makes a selection and theSET payment transaction is launched. The calculated prices are presentedto the customer along with information describing the alternativeversions of the selected video. This information may include theduration of the video, a textual description of the version (e.g.,complete coverage, or highlights). Based on this information, thecustomer may select the specific version he or she wants to buy. Oncethe specific version has been selected the user has to provideelectronic payment information (e.g., credit card information) that canbe used in an electronic transaction. A Secure Electronic Transaction islaunched between the customer and the merchant and between the merchantand a payment gateway (28 of FIG. 1) to verify that the paymentinformation is valid and that credit limits are not exceeded.

[0055] In step 4, the payment transaction is performed successfully. Ifthe payment information is accepted by the payment gateway (28 ofFIG. 1) a payment transaction occurs between the merchant's bank and thebuyer's bank. The merchant server connected to the hypervideo server isinformed about the positive transaction result. The next step is toapply service filters to the selected video to generate the version forwhich payment was cleared. If, however, the payment information was notaccepted by the payment gateway, the rejection is forwarded to thecustomer who is then given the opportunity to make a different selectionor to provide additional payment information. FIG. 9 illustrates thetransaction with sequences 72, video meta-data 73, and filteringmeta-data 74 with hyper video elements 75 that are not part of theselected version and should not be delivered within the initiatedthread.

[0056] In step 5, the filtering process is initiated. If the paymenttransaction was successfully completed the filtering process can beinitiated to determine which content to be delivered to the customer.The customer should only receive hypervideo elements that are part ofthe version for which the payment was made. The filtering process usesfiltering meta-data to determine which NSSs and AIUs should be removedfrom the stream delivered to the customer. Although the physical data isstill available on the hypervideo server, these elements will appear asbeing “removed” from the video that is delivered to the end-user.Referring to the previous example, the hypervideo server would nowdeliver the sequences 76 shown in FIG. 10 to the customer. Sequence “c”has been dropped along with all hyperlinks pointing to AIUs within it.

[0057] In step 6, depending on the payment scheme, the version selectedby the customer might allow for automatic insertion of promotional videosequences 78 into regular sequences 79. The method for selecting thesequences to be inserted and the method for determining where to insertthese sequences will be further discussed below. Referring to theprevious example and assuming that the version selected by the userallows for insertion of promotional video, promotional sequence “p” 78is inserted as illustrated in FIG. 11 showing selected NSSs before thestreaming process starts.

[0058] In step 7, files are copied to the streaming server. The files tobe copied depend on the functionality of the streaming server. No filewill be copied if it is already stored in the streaming server, forinstance, due to a previous request for the same version of the video.If the server does not support playing back a script, as shown in FIG.12a, the sequences and AIUs 80 are reassembled and combined togetherwith promotional sequence inserts 82 to form a streaming file 86 that iscopied to streaming server 81. The video may also be transcoded if thestreaming format is different from the original format of the video orif the hypervideo data could also be transcoded into the properstreaming format (e.g. Microsoft NetShow ASFv2 or Real Network videoplayer format) before being copied to the streaming server 81. On theother hand, if the server supports playing back script files, as shownin FIG. 12b, a presentation script file 87 needs to be copied tostreaming server 81 along with media file(s) 88 containing sequences andAIUs 80 being addressed by the script and promotional sequence file(s)89 containing promotional sequence inserts 82. The presentation scriptfile 87 specifies the locations of sequences and AIUs 80 and promotionalsequence inserts 82 within the media files 88 and promotional sequencefiles 89 and specifies how these elements are to be assembled by thestreaming server. If the streaming server does not support script filedirected streaming, the NSSs and AIUs that passed the filtering stepwill now be reassembled and combined together with promotional sequenceinserts 82 to create the version that was selected by the customer. Thenecessary files are now copied to the streaming server 81. The video mayalso be transcoded if the streaming format is different from theoriginal format of the video or the hypervideo data could also betranscoded into a commercial streaming format (e.g. Microsoft NetShowASFv2 or Real Network video player format) and stored in the streamingserver 81.

[0059] As soon as the filtering process is completed the streamingprocess, step 8 as shown in FIG. 13, can be initiated where streamingapplication thread 83 can be opened between video server 84 and clientbrowser 85. The appropriate streaming player is started within theclient browser to receive and display hypervideo data. The customer mayactivate any of the hyperlinks to AIUs located within the scope of thehypervideo. Also, the customer may activate hyperlinks to AIUs outsideof the scope of the filtered video. This will implicitly start a newprice negotiation and filtering process as described in step 2 above.

[0060] In step 9, an optional step, the merchant may have the streamingprocess monitored to keep exact records of which parts of the selectedhypervideo was actually delivered to the customer. This information canbe used as input to step 11. This step is shown in FIG. 14.

[0061] In step 10, another streaming process with another client isinitiated and started. Assuming that a similar process is initiated byanother client and that streaming sequences A+B+C+D have been selected,the streaming server builds up two different application threads and thestreaming data shown in FIG. 15 is sent over two different connectionsonly one physical connection to the Intranet/Internet service provideris required. The initial video content is the same and is stored onlyonce but, without creating additional video data, the streaming processgenerates in real time two different versions using the same primarydigital video content. If possible, the hypervideo data stored in thestreaming server will also be shared between concurrent sessions orreused by subsequent sessions.

[0062] In step 11, another optional step, the customer is reimbursed forhypervideo elements not consumed if such a payment scheme exists. Thedetailed record from the monitoring done in step 9 will be analyzed toidentify elements not consumed. The accumulated price for these elementsare calculated. A Secure Electronic Transaction (SET) is establishedbetween the merchant and the payment gateway to reimburse the customer.Once the reimbursement has been completed, this information is also sentto the customer.

[0063] The following will provide a detailed description of thefiltering process. This description is divided into three sections, thefirst section describing filtering of narrative sequences, the secondsection describing filtering of hyperlinks, and the third sectiondescribing automatic insertion of promotional video sequences.

[0064] Narrative sequences of scenes filtering is the type of filteringthat is used to define which narrative sequences are being included in aspecific version of a hypervideo. It should be noted that each versionmight freely combine NSSs from the original (unfiltered) hypervideo.Thus, some of the NSSs from the original hypervideo might not be used inany of the filtered versions, some might be used in one filtered versiononly, and some might be shared among several different versions.

[0065] The following will describe basic sequence filtering. A filteringservice defines which parameters are to be used for filtering. Thefiltering meta-data associated with a hypervideo specifies which valuesare given to each NSS for these parameters. The filtering parameters fora hypervideo filtering service may, for instance, be depth_of_coverageand type_of_information. The first parameter, depth_of_coverage, mighttake values complete_coverage, intermediate_coverae, and highlights. Thesecond parameter, type_of_information, might take valuesmanagement_information, sales_information, and technical_information.Each NSS specifies which values each of these parameters takes. If novalue is given for these parameters, the NSS will not be included in anyversion of the hypervideo. If several values are given for theseparameters, the NSS will be included in several versions.

[0066] Also, it is possible for each filtering service to define thetypes of versions it offers. For instance, a filtering service mightdefine three types of versions, gold version, silver version, andstandard version. The filtering service meta-data also specifies whichfiltering parameter values are associated with each of the versions. Thegold version mentioned above, for instance, might take value completecoverage for the depth of coverage parameter and management informationand sales information for the type of information parameter.

[0067] The following will describe hierarchical filtering. As describedby Hjelsvold et al. 1998, a video (and implicitly a narrative sequenceof scenes) can be structured into a hierarchical composition. Atelevision news broadcast, for instance, might be composed of newsstories that again are composed of sub-stories or video shots. In such ahierarchy, the parameters associated with a parent node should alsocover the children. For instance, if a news story is classified as beingrelated to sales information, all sub-stories and shots belonging tothis story would also (implicitly) be classified as sales information.If a filter determines that a node in such a hierarchical compositionshould not be included in a specific version, none of the sub-elementsof the given node will be included either. On the other hand, if thenode is passing the filter but some of the sub-elements do not, only thespecific sub-elements will be removed when creating the filteredversion.

[0068] The following will describe parallel track filtering. If thehypervideo specifies parallel tracks of information to be streamed, thefiltering service will define whether the various tracks are filteredtogether or separately. If the tracks are to be filtered together and anNSS is not to be included in the filtered version of the hypervideo, theinformation that originally was scheduled to be streamed on the othertracks will not be included in the filtered version of these trackseither. If the tracks are to be filtered separately, the other tracksare not affected. Synchronization between the tracks is stillmaintained, however, so no information is delivered to the customer forthe track that has been filtered out for a period of time.

[0069] If the alternative streams can be fed into a specific track, thepayment information may be used to select the specific stream to bedelivered. Again, this depends on the match between the filteringparameter values for the specific version of the hypervideo and theparameter values assigned to each alternative stream.

[0070] The following will describe hyperlink filtering. Hyperlinks allowthe user to follow references from one AIU to another. (The destinationAIU may be any information unit such as a text document, a time pointwithin an NSS or a spatio-temporal part of a NSS.) Before describing thehyperlink filtering in more detail we will describe how some hyperlinksare automatically created when a hypervideo is being prepared for afiltering service.

[0071] The automatic hyperlinking analyzes parameters assigned to eachAIU and analyzes automatic hyperlinking rules stored in the filteringmeta-data to determine if a hyperlink should be automaticallyestablished between a NSS and another NSS or information unit. Assume,for instance, that the merchant offers instructional hypervideos forfix-it-yourself car mechanics. Assume also that the meta-data for allNSS in these hypervideos contains the names of the components beingmentioned or used within the NSS. An automatic hyperlinking rule mightdefine that whenever component is being referred to in the meta-data fora NSS a hyperlink should be created to the white paper for thiscomponent.

[0072] Hyperlinks will be filtered based on the filtering parametervalues associated with the source and the destination AIUs. Thehyperlink can only exist if both the source and the destination AIUspasses the filtering process and are to be included in the filteredversion of the hypervideo. If, on the other hand, one or both of theAIUs do not fulfill the filtering criteria, the hyperlink will not beincluded in the meta-data describing the filtered version.

[0073] With filtering based on link parameters, the meta-data associatedwith each hyperlink might also define the parameter values for whichthis hyperlink should be included. This allows the author of thehypervideo to restrict the possibility to follow certain hyperlinksbased on the selected set of parameter values.

[0074] The following will describe multidimensional hyperlinking usingAIUs. So far, it has been assumed that the hyperlinks connect two AIUstogether, one source and one destination AIU. The filtering meta-datamight also define multiple destinations AIU to be connected to onesource AIU. Each branch of this multiple destination hyperlink should beassociated with filtering meta-data for the system to choose the properdestination AIU once a set of filtering parameter values is selected.FIG. 16 illustrates an example where an AIU is the origin of threehyperlinks associated with different price selections. The filteringprocess will select only the hyperlink that is appropriate for the pricecategory selected by the end-user and include it in the data stream.Thus, the user can but only access one of the defined informationplanes. This example doesn't include any video hyperlink.

[0075]FIG. 17 illustrates an extension to the previous example, FIG. 16,where an AIU is associated with a product demonstrated on the video.This source AIU is again associated with three hyperlinks, one to bechosen for customers buying the premium version of the hypervideo, onefor customers buying the affiliate version of the hypervideo, and onefor standard customers. Also, the AIU contains a hyperlink to anotherNSS which will be sent to the end-user if the end-user selects the AIU.In this case, the same NSS will be shown to the end-user in all threeversions of the hypervideo.

[0076]FIG. 18 illustrates a similar scenario as the previous one, FIG.17. The difference is that the hypervideo is also branching depending onwhich version the customer is viewing therefore showing versionbranching using AIU's.

[0077] The following will describe in detail automatic insertion ofpromotional video sequences. If the user accepts, promotional videosequences (such as TV commercials) will also be inserted into the videoto be delivered. In addition, the end-user may be asked to completequestionnaires or user feedback forms.

[0078] The merchant may statically define which promotional videosequences or questionnaires are to be inserted into the differenthypervideos. In addition, the filtering service may define parametersand selection rules for automatically selecting among availablepromotional video sequences or questionnaires. Alternatively, an off theshelf software package for advertisement selection could be used. Thefiltering service also specifies the amount of promotional video thatcan be inserted for a given version and the number of questions in thequestionnaire. The filtering service meta-data is analyzed to findpromotional video sequences and questionnaires that best matches theselection rules without overflowing the amount limits.

[0079] Again, the merchant may statically indicate the places in ahypervideo where promotional video might be inserted. Also, thefiltering service may define parameters and insertion rules forautomatically selecting the places where promotional video orquestionnaires can be-inserted. For-instance, a filtering service fortelevision news might define that promotional video sequences shouldonly be inserted between news stories, that each “block” of promotionalvideo should be no more than one minute long, and that the insertionpoints should be distributed equally over the course of the news. Also,the service might define that questionnaires should be presented betweennew stories, but as near the middle of the filtered video as possible.

[0080] The following will describe different purchasing models andpayment schemes that will be used to compute the price for a selectedversion of a hypervideo. A distinction could be made between twodifferent purchasing models into which the filtering and informationdelivery could be applied. The first enables a customer to subscribe todifferent qualities of information (for instance various levels ofaccess to information provided by a call center or support center); thesecond allows the customer to select and purchase hypervideo informationon-demand just like buying newspapers from a newsstand.

[0081] A subscription model assumes that the hypervideo merchant and thecustomer approve a subscription contract. This contract defines thelifetime of the subscription and renewal procedures. It also describesthe payment options (payment parameters and billing information). Inthis model, end-users have to provide identification information (forinstance account name and password) for the system to retrieve thepre-stored filtering parameters. The user will not have to or be able tomake a price selection once the identity is verified. The pre-storedparameters will be used as input to the filtering. There are two waysfor a subscriber to purchase a hypervideo using a different paymentoption than the one defined by the current subscription. First, the usermay decide to permanently upgrade or degrade the subscription. Second,the user may make a purchase following the newsstand model. Thesubscription model generates a much smaller number of paymenttransactions than the newsstand model. Thus, it improves the overallefficiency.

[0082] A newsstand model defines a more dynamic way to associatefiltering and payment transaction. It is a case-by-case solution, wherefiltering parameters are determined for the current delivery sessiononly. In this model, the customer will have to make a price selectionfor each hypervideo he or she wants to purchase. This model allowsend-users to make casual and spontaneous purchases without having tosubscribe to a service.

[0083] The following will describe multi-level pricing. In the simplestcase, the price is independent from the actual video. The filter serviceexample presented earlier (FIGS. 16-18) specified three types ofhypervideo versions, the gold version, the silver version, and thestandard version. The merchant may in this case define the price classto be used for each of these types of videos. Thus, for instance, thegold version of any of the videos may cost $10, the silver version $6,while the standard version may cost only $2.

[0084] Alternatively, the filtering meta-data may contain paymentparameters that are associated with individual NSSs. In this case, theprice would be computed based on the set of NSSs that are included in agiven version. These parameters specify a price class that can later beconverted to an exact price. Three different price elements may be usedto compute the price of a given video in this case. One price element isthe price class associated with the hypervideo itself, allowing eachindividual version of a hypervideo to be associated with a specificprice class. This allows a filtered version of one hypervideo to have aprice different from the same version of a different hypervideo.

[0085] The next price element is accumulating the price contributionsfrom the sub-elements of the hypervideo. For instance, in a televisionnews filtering system, each news story might be given a specific priceclass, and the price for a filtered version might be the computed byadding the price contributions for each of the stories together.

[0086] Another price element is related to the duration of the filteredvideo. Each NSS may be associated with a price class that would bemapped to a price per unit time. The price for the NSS would be computedby multiplying the price per unit time with the duration of the NSS.

[0087] The following will describe prices as a function of time. Allprice elements might also depend on the time at which the video wasrequested. The time varying price function has two elements. Theabsolute part defines the functional dependency between the price andthe day and time of the request. It may, for instance, be more expensiveto buy a financial news story in the middle of a working day than onSunday morning. The relative part defines how the price changes relativeto the age of the information object. A financial news story, forinstance, may have a higher price immediately after being published thantwo days later.

[0088] For discount pricing, all price elements might be given negativevalues, for instance, to encourage customers to select versionscontaining promotional or advertising information.

[0089]FIG. 19 describes refunding for hypervideo segments not consumed.In some business models, it would be necessary to refund the user forthe parts of the ordered hypervideo that was not consumed if theend-user cancels the streaming process for some reason (such asconnection failure, personal user's wish to break the session, unsuitedcontent etc.).

[0090] If the payment scheme being used specifies that the end-usershould receive such a refund, the delivery process will be monitored anda detailed usage record will be generated. When the end-user closes thevideo transfer session, the usage record will be consolidated andcompared to the filtered video that the end-user ordered to determinewhich parts of the video the end-user did not consume. The system willthen compute the corresponding value and refund this amount.

[0091] In summary, the present invention is a system for givingcustomers access to a subset of hypervideo items (a version) stored on amerchant's server based on a selected payment. The system does notrequire the use of encryption and sales of decryption keys. The systemalso allows the merchant to define parameters to be used for creatingthe various versions and for computing the price of the given version.

[0092] It is not intended that this invention be limited to the hardwareor software arrangement, or operational procedures shown disclosed. Thisinvention includes all of the alterations and variations thereto asencompassed within the scope of the claims as follows.

We claim:
 1. A system for hypervideo filtering based on end-user paymentinterest and capability comprising: a filter server for interfacing witha client; and a service administration system connected to said filterserver.
 2. A system for hypervideo filtering based on end-user paymentinterest and capability as claimed in claim 1 wherein said filter servercomprises: a www server for interfacing with said client; a commerceserver connected to said www server; a SET server connected to saidcommerce server; a database server connected between said www server andsaid service administration system; a filtering meta-data databaseconnected to said database server; a media data server connected to saidwww server; an archiving system connected to said media data server andsaid database server; and a streaming server connected to said mediadata server.
 3. A system for hypervideo filtering based on end-userpayment interest and capability as claimed in claim 1 wherein saidservice administration system comprises: management tools connected tosaid filter server; and a meta-data editing tool connected to saidfilter server.
 4. A system for hypervideo filtering based on end-userpayment interest and capability as claimed in claim 2 wherein said wwwserver comprises: a media file generator connected to said media dataserver and said database server; and a filtering process connected tosaid commerce server and said database server.
 5. A system forhypervideo filtering based on end-user payment interest and capabilityas claimed in claim 2 wherein said archiving system comprises: ameta-data database connected to said database server; and a media datarepository connected to said media data server.
 6. A system forhypervideo filtering based on end-user payment interest and capabilityas claimed in claim 2 wherein said SET server connects to a clearinghouse comprising: a payment gateway.
 7. A system for hypervideofiltering based on end-user payment interest and capability comprising:a filter server for interfacing with a client; a service administrationsystem connected to said filter server; and an archiving systemconnected to said filter server.
 8. A system for hypervideo filteringbased on end-user payment interest and capability as claimed in claim 7wherein said filter server comprises: a www server for interfacing withsaid client; a commerce server connected to said www server; a SETserver connected to said commerce server; a database server connectedbetween said www server and said service administration system; afiltering meta-data database connected to said database server; a mediadata server connected to said www server; and a streaming serverconnected to said media data server.
 9. A system for hypervideofiltering based on end-user payment interest and capability as claimedin claim 7 wherein said service administration system comprises:management tools connected to said filter server; and a meta-dataediting tool connected to said filter server.
 10. A system forhypervideo filtering based on end-user payment interest and capabilityas claimed in claim 7 wherein said archiving system comprises: ameta-data database connected to said filter server; and a media datarepository connected to said filter server.
 11. A system for hypervideofiltering based on end-user payment interest and capability as claimedin claim 8 wherein said www server comprises: a media file generatorconnected to said media data server and said database server; and afiltering process connected to said commerce server and said databaseserver.
 12. A system for hypervideo filtering based on end-user paymentinterest and capability as claimed in claim 8 wherein said SET serverconnects to a clearing house.
 13. A method for hypervideo filteringbased on end-user payment interest and capability comprising the stepsof: digitizing a video tape into raw video data; determining videoshots, scenes and NSSs; generating meta-data to define NSS sequencing,to identify AIUs and to add hyperlinks between said AIUs; and filteringsaid meta-data.
 14. A method for hypervideo filtering based on end-userpayment interest and capability as claimed in claim 13 wherein filteringsaid meta-data comprises the steps of: selecting said NSSs or videos tobe viewed; calculating price according to payment or customer profile;performing a payment transaction; initiating a filtering process;copying a presentation script -file and-media files comprising data usedby said presentation script file to a streaming server; and initiating astreaming process.
 15. A method for hypervideo filtering based onend-user payment interest and capability as claimed in claim 13 whereinfiltering said meta-data comprises the steps of: selecting said NSSs orvideos to be viewed; calculating price according to payment or customerprofile; performing a payment transaction; initiating a filteringprocess; copying a streaming file to a streaming server; and initiatinga streaming process.
 16. A method for hypervideo filtering based onend-user payment interest and capability as claimed in claim 14 whereinfiltering said meta-data further comprises the steps of: inserting apromotional video; monitoring a streaming process; and providingreimbursement for hypervideo elements not consumed.
 17. A method forhypervideo filtering based on end-user payment interest and capabilityas claimed in claim 13 wherein filtering said meta-data comprises thesteps of: narrative scenes filtering; and hyperlink filtering.
 18. Amethod for hypervideo filtering based on end-user payment interest andcapability as claimed in claim 17 wherein narrative scenes filteringcomprises the steps of: basic sequence filtering; hierarchicalfiltering; and parallel track filtering.
 19. A method for hypervideofiltering based on end-user payment interest and capability as claimedin claim 17 wherein hyperlink filtering comprises the steps of:automatic hyperlinking; AIU parameter filtering; link parameterfiltering; multidimensional hyperlinking using said AIUs; and versionbranching using said AIUs.
 20. A method for hypervideo filtering basedon end-user payment interest and capability as claimed in claim 16wherein inserting a promotional video comprises the steps of: selectinga promotional sequence for insertion into said hypervideo structure; andselecting an insert point for said promotional sequence.
 21. A systemfor hypervideo filtering based on end-user payment interest andcapability comprising: a filter server for interfacing with a client;and a service administration system connected to said filter server;wherein said filter server comprises: a www server for interfacing withsaid client; a commerce server connected between said www server andsaid client; a SET server connected to said commerce server; a databaseserver connected to said www server; a media data server connected tosaid www server; a data database connected to said database server andsaid media data server; and a streaming server connected between saidmedia data server and said client.